Generating a reference digital image based on an indicated time frame and searching for other images using the reference digital image

ABSTRACT

A digital image and a text string is received. The text string can be processed to identify at least a time frame and determine whether the time frame is a future time frame or a past time frame. How at least one element of the first digital image will change or has changed during the time frame can be predicted. At least one reference digital image can be generated, the reference digital image including at least one change to the at least one element corresponding to how the at least one element will change or has changed during the time frame. The reference digital image to each of a plurality of other digital images. A correlation parameter can be assigned to each of the plurality of other digital images. A portion of the plurality of other digital images having highest correlation parameters can be output for presentation to a user.

BACKGROUND

The present invention relates to information retrieval.

Information retrieval is the activity of obtaining information resourcesrelevant to an information need from a collection of informationresources. Searches can be based on metadata, text and image indexing.Automated information retrieval systems are used to facilitate searchingand reduce information overload. Many universities and public librariesuse information retrieval systems to provide access to books, journalsand other documents. Web search engines are the most visible informationretrieval applications.

SUMMARY

A method includes receiving a first digital image and receiving a textstring indicating at least a time frame. The method also can includeprocessing the text string to identify at least the time frame anddetermine whether the time frame is a future time frame or a past timeframe. The method also can include, responsive to determining whetherthe time frame is the future time frame or the past time frame,predicting how at least one element of the first digital image willchange or has changed during the time frame. The method also caninclude, responsive to predicting how the at least one element of thefirst digital image will change or has changed during the time frame,generating, using a processor, at least one reference digital image thatis a revised version of the first digital image, the reference digitalimage including at least one change to the at least one elementcorresponding to how the at least one element will change or has changedduring the time frame. The method also can include comparing thereference digital image to each of a plurality of other digital images.The method also can include, based on comparing the reference digitalimage to each of the plurality of other digital images, assigning acorrelation parameter to each of the plurality of other digital images,each correlation parameter indicating how closely a respective otherdigital image matches the reference digital image. The method also caninclude outputting for presentation to a user a portion of the pluralityof other digital images having highest correlation parameters.

A system includes a processor programmed to initiate executableoperations. The executable operations include receiving a first digitalimage and receiving a text string indicating at least a time frame. Theexecutable operations also can include processing the text string toidentify at least the time frame and determine whether the time frame isa future time frame or a past time frame. The executable operations alsocan include, responsive to determining whether the time frame is thefuture time frame or the past time frame, predicting how at least oneelement of the first digital image will change or has changed during thetime frame. The executable operations also can include, responsive topredicting how the at least one element of the first digital image willchange or has changed during the time frame, generating at least onereference digital image that is a revised version of the first digitalimage, the reference digital image including at least one change to theat least one element corresponding to how the at least one element willchange or has changed during the time frame. The executable operationsalso can include comparing the reference digital image to each of aplurality of other digital images. The executable operations also caninclude, based on comparing the reference digital image to each of theplurality of other digital images, assigning a correlation parameter toeach of the plurality of other digital images, each correlationparameter indicating how closely a respective other digital imagematches the reference digital image. The executable operations also caninclude outputting for presentation to a user a portion of the pluralityof other digital images having highest correlation parameters.

A computer program includes a computer readable storage medium havingprogram code stored thereon. The program code is executable by aprocessor to perform a method. The method includes receiving, by theprocessor, a first digital image and receiving a text string indicatingat least a time frame. The method also can include processing, by theprocessor, the text string to identify at least the time frame anddetermine whether the time frame is a future time frame or a past timeframe. The method also can include, responsive to determining whetherthe time frame is the future time frame or the past time frame,predicting, by the processor, how at least one element of the firstdigital image will change or has changed during the time frame. Themethod also can include, responsive to predicting how the at least oneelement of the first digital image will change or has changed during thetime frame, generating, by the processor, at least one reference digitalimage that is a revised version of the first digital image, thereference digital image including at least one change to the at leastone element corresponding to how the at least one element will change orhas changed during the time frame. The method also can includecomparing, by the processor, the reference digital image to each of aplurality of other digital images. The method also can include, based oncomparing the reference digital image to each of the plurality of otherdigital images, assigning, by the processor, a correlation parameter toeach of the plurality of other digital images, each correlationparameter indicating how closely a respective other digital imagematches the reference digital image. The method also can includeoutputting, by the processor, for presentation to a user a portion ofthe plurality of other digital images having highest correlationparameters.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of a computingenvironment.

FIG. 2 is a flow chart illustrating an example of a method of rankingdigital images.

FIG. 3 is a block diagram illustrating example architecture for in imagesearch system.

DETAILED DESCRIPTION

This disclosure relates to information retrieval systems and, moreparticularly, to an image search system. The image search system canfacilitate people searches using digital images and text stringsindicating search criteria.

In accordance with the present arrangements, a user can communicate toan image search system a digital image and a text string includingsearch criteria. The search criteria can be provided in a human languageand can be generated using a keyboard or by performing speechrecognition on spoken utterances provided by the user. For example, theuser can provide a text string that states “Bella attended BostonUniversity with me 20 years ago.” The image search system can processthe text string to identify search criteria. In this example, the searchcriteria can include the name of a person, “Bella,” and a specificlocation, “Boston University.” The search criteria also can indicate apredicate and a time frame. In this example, the text “20” can indicatethe time frame. The word “attended” is a predicate that, based on thestem “ed,” indicates that the time frame is in the past. The word “ago”is an adverb that also can indicate that the time frame is in the past.

Based on the predicate and the time frame, the image processing systemcan process the digital image to identify a person depicted in theimage, and predict how visual features of that person have changed overthe time frame (e.g., over the last 20 years). The image processingsystem can apply corresponding changes to the visual features togenerate, from the digital image, a reference digital image includingsuch changes. For example, the reference digital image can depict howthe person may presently look, or how the person may have looked in thepast. Using the reference digital image, the image processing system canuse facial recognition to search for other digital images that maydepict the person, for example more recent images or older images. Theimage processing system can determine a level of correlation of each ofthe other images to the reference digital images, and output forpresentation to the user the other images having the highest levels ofcorrelation.

Several definitions that apply throughout this document now will bepresented.

As defined herein, the term “digital image” means a numericrepresentation (e.g., a binary representation) of an image visuallydepicting at least one person or a drawing visually depicting at leastone person. Examples of a digital image include, but are not limited to,a vector image, a raster image and a bitmapped image. A digital imagecan be generated by digitally scanning a photograph or capturing theimage with a digital image capture device, for example a digital camera,smart phone, or the like.

As defined herein, the term “revised version of a digital image” means aversion of a digital image that has been revised by processing thedigital image, using a processor, to change at least one element of thedigital image.

As defined herein, the term “element of a digital image” means an itemand/or feature of an image visually depicted by a digital image.

As defined herein, the term “text string” means one or more phrases,clauses and/or sentences that are written or spoken in a human language.

As defined herein, the term “human language” is a language spoken orwritten by human beings that is not a computer programming language. A“human language” may be referred to as a “natural language.”

As defined herein, the term “predicate” means a part of a text stringthat expresses what is said of a subject of the text string and thatconsists of a verb with or without objects, complements and/or adverbialmodifiers. A predicate may include a stem indicating a meaning of thepredicate, for example whether the predicate infers a past time frame ora future time frame.

As defined herein, the term “stem” means a part of an inflected wordthat remains after the inflected part of the word is removed. A stem canserve as a canonical indicator of a predicate's meaning.

As defined herein, the term “correlation parameter” means a parameterindicating a level of correlation between a reference digital image andanother digital image.

As defined herein, the term “time frame” means a period of time. A timeframe can be specified as a number of months and/or years, or can beindicated using a day, week, year or month. For example, a year 1995 canindicate a time frame from the year 1995 until present. A year 2035 canindicate a time frame from the present until the year 2035.

As defined herein, the term “client device” means a processing systemincluding at least one processor and memory that requests sharedservices from a server, and with which a user directly interacts.Examples of a client device include, but are not limited to, aworkstation, a desktop computer, a mobile computer, a laptop computer, anetbook computer, a tablet computer, a smart phone, a personal digitalassistant, a smart watch, smart glasses, a gaming device, a set-top box,a smart television, and the like. Network infrastructure, such asrouters, firewalls, switches, access points, and the like, are notclient devices as the term “client device” is defined herein.

As defined herein, the term “social networking system” means a systemthat provides social networking services, for instance via one or morewebsites. A social networking service is an online service platform onwhich social networks or social relations are built among people who,for example, share interests, activities, backgrounds or real-lifeconnections, and via which people can share information with oneanother. As the term “social networking system” is defined herein, amessaging system per se (e.g., an e-mail system, a text messagingsystem, or an instant messaging system) is not a social networkingsystem, though a social networking system can include an e-mailmessaging system, a text messaging system and/or an instant messagingsystem in addition to one or more social networking components.

As defined herein, the term “natural language analysis” means a processthat derives a computer understandable meaning of a human language.

As defined herein, the term “responsive to” means responding or reactingreadily to an action or event. Thus, if a second action is performed“responsive to” a first action, there is a causal relationship betweenan occurrence of the first action and an occurrence of the secondaction, and the term “responsive to” indicates such causal relationship.

As defined herein, the term “computer readable storage medium” means astorage medium that contains or stores program code for use by or inconnection with an instruction execution system, apparatus, or device.As defined herein, a “computer readable storage medium” is not atransitory, propagating signal per se.

As defined herein, the term “processor” means at least one hardwarecircuit (e.g., an integrated circuit) configured to carry outinstructions contained in program code. Examples of a processor include,but are not limited to, a central processing unit (CPU), an arrayprocessor, a vector processor, a digital signal processor (DSP), afield-programmable gate array (FPGA), a programmable logic array (PLA),an application specific integrated circuit (ASIC), programmable logiccircuitry, and a controller.

As defined herein, the term “real time” means a level of processingresponsiveness that a user or system senses as sufficiently immediatefor a particular process or determination to be made, or that enablesthe processor to keep up with some external process.

As defined herein, the term “output” means storing in memory elements,writing to display or other peripheral output device, sending ortransmitting to another system, exporting, or the like.

As defined herein, the term “automatically” means without userintervention.

As defined herein, the term “user” means a person (i.e., a human being).

FIG. 1 is a block diagram illustrating an example of a computingenvironment 100. The computing environment 100 can include an imagesearch system 110, at least one client device 130 and a plurality ofimage sources 150, 152, 154. The image sources 150-154 can be, forexample, websites, social networking systems, public record systems,and/or the like.

The image search system 110 can be communicatively linked to the clientdevice 130 and the plurality of image sources 150-154 via one or morecommunication networks 140. A communication network 140 is a medium usedto provide communications links between various devices and dataprocessing systems connected together within the computing environment100. The communication network 140 may include connections, such aswire, wireless communication links, or fiber optic cables Thecommunication network 140 can be implemented as, or include, any of avariety of different communication technologies such as a WAN, a LAN, awireless network, a mobile network, a Virtual Private Network (VPN), theInternet, the Public Switched Telephone Network (PSTN), or the like.

The image search system 110 can be implemented as one or more processingsystems, each including at least one processor and memory elements, andcan include a user interface 112, a text analyzer 114, an imageprocessor 116 and an image search engine 118. A user of the clientdevice 130 can log into and interact with the image search system 110via the user interface 112. For example, via the user interface 112, theuser can, using the client device 130, upload a text string 170 and adigital image 172 to the image search system 110. The digital image 172can be, for example, a digital scan of a photograph or an image capturedwith a digital image capture device, for example a digital camera, smartphone or the like. The text string 170 can be generated by the user in ahuman language, and include search criteria to be used by the imagesearch system 110 to search for other digital images 160, 162, 164correlating to the digital image, as will be described. The searchcriteria can indicate a name of a person depicted in the digital image172. The search criteria also can indicate a location, such as a place,an address, a city, a county, a state, a province, a country, and/or thelike where the person is known, or was known, to have lived, workedvisited, etc., or where the photograph or image was captured.

Further, the search criteria can include a predicate and a time frame.The predicate and the time frame can indicate the nature of the digitalimage search requested by the user. For example, the predicate and timeframe can indicate that the digital image 172 was captured some time ago(e.g., five years ago, ten years ago, twenty years ago, etc.), on aparticular date, or within a particular week, month and/or year, andother search criteria contained in the text string 170 can indicate thatthe user desires to see more recent digital images 160-164 depicting aperson that is depicted in the digital image 172. In another example,the search criteria contained in the text string 170 can indicate thatthe digital image 172 was recently captured, and the predicate and timeframe can indicate that the user desires to see older digital images160-164 depicting a person that is depicted in the digital image 172,for instance digital images captured five years ago, ten years ago,twenty years ago, etc., on a particular date, or in a particular week,month and/or year.

Responsive to receiving the text string 170, the image search system 110can communicate the text string 170 to the text analyzer 114, which canprocess the text string 170, in real time, to identify the searchcriteria. Such processing can include identifying a name contained inthe text string 170, location information, a time frame, etc. Suchprocessing also can include, based on the predicate contained in thetext string 170, determining whether the time frame indicated in thetext string 170 is a future time frame or a past time frame. To identifythe search criteria contained in the text string 170, the text analyzer114 can implement natural language processing (NLP) and semanticanalysis on information contained in text string 170. NLP is a field ofcomputer science, artificial intelligence and linguistics whichimplements computer processes to facilitate interactions betweencomputer systems and human (natural) languages. NLP enables computers toderive computer-understandable meaning from natural language input. TheInternational Organization for Standardization (ISO) publishes standardsfor NLP, one such standard being ISO/TC37/SC4. Semantic analysis is theimplementation of computer processes to generate computer-understandablerepresentations of natural language expressions. Semantic analysis canbe used to construct meaning representations, semanticunderspecification, anaphora resolution, presupposition projection andquantifier scope resolution, which are known in the art. Semanticanalysis is frequently used with NLP to derive computer-understandablemeaning from natural language input. An unstructured informationmanagement architecture (UIMA), which is an industry standard forcontent analytics, may be used by the text analyzer 114 to implement NLPand semantic analysis.

The image processor 116 can receive from the image search system 110 thedigital image 172 and receive from the text analyzer 114 the identifiedsearch criteria. The image processor 116 can process the digital image172, in real time, in accordance with the search criteria to generate areference digital image 180. For example, the image processor 116 canidentify one or more elements in the digital image 172 depicting aperson. Based on the predicate and time frame contained in the searchcriteria, the image processor 116 can process the element(s) to predicthow the elements will change or have changed over the time frame.Whether the time frame is a past time frame or future time frame can beindicated by the predicate. While generating the reference digital image180, the image processor 116 can apply such changes to the elements todepict the elements in the reference digital image 180 with the changesapplied.

By way of example, in the case that an element contained in the digitalimage 172 is a face, the image processor 116 can process the element todetermine an approximate age of the person and determine how features ofthe face will change, or have changed, over the time frame. Examples ofsuch changes include, but are not limited to, shape of the face, skintexture changes, and the like. Further, the image processor 116 candetermine hair color changes. Having determined such changes, the imageprocessor 116 can apply the changes to the elements to generate thereference digital image 180. For example, the image processor 116 candepict the elements as they would have appeared a certain number ofmonths and/or years ago, or how the elements will appear a certainnumber of months and/or years in the future. Prediction of how elementswill change, or have changed, over a given time frame is known to thoseof ordinary skill in the art.

In one non-limiting arrangement, the image processor 116 can be externalto the image search system 110. For example, the image search system 110can include a web-based client application 120 configured to interfacewith an external image processor hosted by a remote system. In such anarrangement, the web-based client application 120 can automaticallycommunicate the digital image 172 and criteria for processing thedigital image 172 to the remote system. The remote system can processthe digital image 172 in accordance with the criteria to generate thereference digital image 180. The criteria can be based on theaforementioned search criteria and can indicate how the digital image172 is to be aged, for example to depict elements as they would haveappeared a certain number of months and/or years ago, or how theelements will appear a certain number of months and/or years in thefuture. Responsive to generating the reference digital image 180, theremote system can communicate the reference digital image 180 to theimage search system 110. For instance, the web-based client application120 can be configured to automatically retrieve the reference digitalimage 180 from the remote system. An example of such a remote system isa system hosting the website http://in20 years.com. The web-based clientapplication 120 can be configured to automatically interface with such awebsite.

Responsive to the reference digital image 180 being generated by theimage search engine 118 or being received from a remote system, theimage search engine 118 can store the reference digital image 180 to acomputer readable storage medium. The image search engine 118 also can,in real time, perform facial recognition on the reference digital image180 to generate corresponding facial recognition data. Generating facialrecognition data based an image is known to those of ordinary skill inthe art.

The image search engine 118 can, in real time, interface with the imagesources 150-154 to retrieve a plurality of digital images 160-164 thatpotentially depict the same person depicted in the reference digitalimage 180. To retrieve the digital images 160-164, the image searchengine 118 can query the image sources 150-154 using the facialrecognition data and/or the identified search criteria. In illustration,the search criteria can specify a name of a person indicated in the textstring 170, a location indicated in the text string 170, and/or thelike. The image sources 150-154 can identify digital images 160-164corresponding to the facial recognition data and/or search criteria, andcommunicate to the image search system 110 such digital images 160-164and/or facial recognition data for each of such digital images 160-164.In the case that the image search system 110 receives the digital images160-164 without facial recognition data, the image search engine 118 canperform facial recognition on the received digital images 160-164 togenerate, for each received digital image 160-164, respective facialrecognition data.

Regardless of whether the image search system 110 receives the facialrecognition data for the digital images 160-164 from the image sources150-154 or the image search system 110 generates such facial recognitiondata, the image search engine 118 can, in real time, compare thereference digital image 180 to the digital images 160-164 by comparingthe facial recognition data corresponding to the reference digital image180 to the facial recognition data for each of the received digitalimages 160-164. Moreover, the image search system 110 can assign acorrelation parameter to each of the digital images 160-164. Eachcorrelation parameter can indicate how closely a respective digitalimage 160-164 matches the reference digital image 180. For example, fora particular digital image 160-164, the correlation parameter assignedto that digital image 160-164 can be determined based on a level ofcorrelation of the facial recognition data for that digital image160-164 to the facial recognition data for the reference digital image180.

Based on the correlation parameters assigned to the respective digitalimages 160-164, the image search engine 118 can, in real time, identifyone or more digital images 160-164 having the highest correlationparameter(s), and select that/those digital image(s) 160-164. Forexample, the image search engine 118 can select the one digital image160-164 having the highest correlation parameter. In another example,the image search engine 118 can select one, two, three, four, five, ten,fifteen, twenty, thirty, forty, fifty, etc. of the digital images160-164 having the highest correlation parameters. In the case that theimage search engine 118 has not yet received the selected digital images160-164 from the image sources 150-154, the image search engine 118 canretrieve such digital images 160-164 from the image sources 150-154.Regardless of when the selected digital images 160-164 are retrieved,the image search engine 118 can assign a ranking to each of the selecteddigital images 160-164 a ranking based on the respective correlationparameter. The digital image 160-164 having the highest correlationparameter can be assigned the highest ranking, the digital image 160-164having the next highest correlation parameter can be assigned the nexthighest ranking, and so on.

The image search engine 118 further can access available data for eachof the selected digital images 160-164. In illustration, for eachselected digital image 160-164, the image search engine 118 can, in realtime, query the image source 150-154 from which the digital image160-164 is obtained for data corresponding to the digital image 160-164,for example one or more names associated with the digital image 160-164,one or more locations associated with the digital image, etc. Such datacan be associated with the digital image 160-164 in any suitable manner,for example as one or more tags applied to the digital image 160-164, asmetadata applied to the digital image 160-164, or using a data record(e.g., a record in a database table that identifies the digital image160-164 and the data).

The image search engine 118 can, in real time, output to the clientdevice 130, for presentation to the user, the selected digital images160-164. The image search engine 118 can output the selected digitalimages 160-164 via the user interface 112. In illustration, the imagesearch engine 118 can communicate the selected digital images 160-164 tothe client device 130 as potentially relevant digital images 190. Theimage search engine 118 can include with the potentially relevantdigital images 190 the rankings and/or correlation parameters assignedto each of the selected digital images 160-164. In addition, for each ofsuch digital images 160-164, the image search engine 118 can communicateto the client device 130 the corresponding data accessed for thatdigital image 160-164.

Responsive to receiving the potentially relevant digital images 190 andthe corresponding data, the client device 130 can, in real time, presentthe potentially relevant digital images 190, along with thecorresponding data, to the user of the client device 130, for example ona display. The client device 130 can indicate the ranking and/orcorrelation parameter assigned to each of the potentially relevantdigital images 190. In one arrangement, the client device 130 canpresent the potentially relevant digital images 190 in an order based onthe respective rankings and/or correlation parameters.

FIG. 2 is a flow chart illustrating an example of a method 200 ofranking digital images. The method 200 can be implemented by the imagesearch system 110 of FIG. 1.

At step 202, the image search system 110 can authenticate a user logginginto the image search system, for example via the user interface 112. Atstep 204, the image search system 110 can receive, via the userinterface 112, from the user a first digital image 172 and a text string170 indicating at least a time frame. The user can communicate the firstdigital image 172 and the text string 170 to the image search system 110using the client device 130.

At step 206, the text analyzer 114 can process the text string toidentify at least the time frame and determine whether the time frame isa future time frame or a past time frame. At step 208, responsive to thetext analyzer 114 determining whether the time frame is a future timeframe or a past time frame, the image processor 116 can predict how atleast one element of the first digital image 172 will change or haschanged during the time frame. At step 210, responsive to the imageprocessor 116 predicting how the at least one element of the firstdigital image will change or has changed during the time frame, theimage processor 116 can generate, using a processor, at least onereference digital image 180 that is a revised version of the firstdigital image. The reference digital image 180 can include at least onechange to the at least one element corresponding to how the at least oneelement will change or has changed during the time frame.

At step 212, the image search engine 118 can compare the referencedigital image 180 to each of a plurality of other digital images160-164. At step 214, based on comparing the reference digital image 180to each of the plurality of other digital images 160-164, the imagesearch engine 118 can assign a correlation parameter to each of theplurality of other digital images 160-164. Each correlation parametercan indicate how closely a respective other digital image 160-164matches the reference digital image 180. At step 216, the image searchengine 118 can output, via the user interface 112, for presentation tothe user a portion of the plurality of other digital images 160-164having highest correlation parameters. In illustration, via the userinterface 112, the image search engine 118 can communicate the portionof the plurality of other digital images 160-164 having highestcorrelation parameters to the client device 130 for presentation to theuser by the client device 130.

FIG. 3 is a block diagram illustrating example architecture for theimage search system 110 of FIG. 1.

The image search system 110 can include at least one processor 305(e.g., a central processing unit) coupled to memory elements 310 througha system bus 315 or other suitable circuitry. As such, the image searchsystem 110 can store program code within the memory elements 310. Theprocessor 305 can execute the program code accessed from the memoryelements 310 via the system bus 315. It should be appreciated that theimage search system 110 can be implemented in the form of any systemincluding a processor and memory that is capable of performing thefunctions and/or operations described within this specification. Forexample, the image search system 110 can be implemented as a server, aplurality of communicatively linked servers, or the like.

The memory elements 310 can include one or more physical memory devicessuch as, for example, local memory 320 and one or more bulk storagedevices 325. Local memory 320 refers to random access memory (RAM) orother non-persistent memory device(s) generally used during actualexecution of the program code. The bulk storage device(s) 325 can beimplemented as a hard disk drive (HDD), solid state drive (SSD), orother persistent data storage device. The image search system 110 alsocan include one or more cache memories (not shown) that providetemporary storage of at least some program code in order to reduce thenumber of times program code must be retrieved from the bulk storagedevice 325 during execution.

One or more network adapters 330 also can be coupled to image searchsystem 110 to enable the image search system 110 to become coupled toclient devices, other systems, computer systems, remote printers, and/orremote storage devices through intervening private or public networks.Modems, cable modems, transceivers, and Ethernet cards are examples ofdifferent types of network adapters 330 that can be used with the imagesearch system 110.

As pictured in FIG. 3, the memory elements 310 can store the componentsof the image search system 110, namely the user interface 112, the textanalyzer 114, the image processor 116, the image search engine 118, andthe reference digital images 180. Optionally, the memory elements 310also can store the web-based client application 120. Being implementedin the form of executable program code, the user interface 112, textanalyzer 114, image processor 116, image search engine 118, andweb-based client application 120 can be executed by the image searchsystem 110 and, as such, can be considered part of the image searchsystem 110. The reference digital images 180 can be generated and storedby the image search system 110 and, thus, also can be considered part ofthe image search system 110. Moreover, the user interface 112, the textanalyzer 114, the image processor 116, the image search engine 118, theweb-based client application 120 and the reference digital images 180are functional data structures that impart functionality when employedas part of the image search system 110.

While the disclosure concludes with claims defining novel features, itis believed that the various features described herein will be betterunderstood from a consideration of the description in conjunction withthe drawings. The process(es), machine(s), manufacture(s) and anyvariations thereof described within this disclosure are provided forpurposes of illustration. Any specific structural and functional detailsdescribed are not to be interpreted as limiting, but merely as a basisfor the claims and as a representative basis for teaching one skilled inthe art to variously employ the features described in virtually anyappropriately detailed structure. Further, the terms and phrases usedwithin this disclosure are not intended to be limiting, but rather toprovide an understandable description of the features described.

For purposes of simplicity and clarity of illustration, elements shownin the figures have not necessarily been drawn to scale. For example,the dimensions of some of the elements may be exaggerated relative toother elements for clarity. Further, where considered appropriate,reference numbers are repeated among the figures to indicatecorresponding, analogous, or like features.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a,” “an,” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “includes,”“including,” “comprises,” and/or “comprising,” when used in thisdisclosure, specify the presence of stated features, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

Reference throughout this disclosure to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment described within this disclosure.Thus, appearances of the phrases “in one embodiment,” “in anembodiment,” and similar language throughout this disclosure may, but donot necessarily, all refer to the same embodiment.

The term “plurality,” as used herein, is defined as two or more thantwo. The term “another,” as used herein, is defined as at least a secondor more. The term “coupled,” as used herein, is defined as connected,whether directly without any intervening elements or indirectly with oneor more intervening elements, unless otherwise indicated. Two elementsalso can be coupled mechanically, electrically, or communicativelylinked through a communication channel, pathway, network, or system. Theterm “and/or” as used herein refers to and encompasses any and allpossible combinations of one or more of the associated listed items. Itwill also be understood that, although the terms first, second, etc. maybe used herein to describe various elements, these elements should notbe limited by these terms, as these terms are only used to distinguishone element from another unless stated otherwise or the contextindicates otherwise.

The term “if” may be construed to mean “when” or “upon” or “in responseto determining” or “in response to detecting,” depending on the context.Similarly, the phrase “if it is determined” or “if [a stated conditionor event] is detected” may be construed to mean “upon determining” or“in response to determining” or “upon detecting [the stated condition orevent]” or “in response to detecting [the stated condition or event],”depending on the context.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A method comprising: receiving a first digitalimage; receiving a text string indicating at least a time frame;processing the text string to identify at least the time frame anddetermine whether the time frame is a future time frame or a past timeframe; responsive to determining whether the time frame is the futuretime frame or the past time frame, predicting how at least one elementof the first digital image will change or has changed during the timeframe; responsive to predicting how the at least one element of thefirst digital image will change or has changed during the time frame,generating, using a processor, at least one reference digital image thatis a revised version of the first digital image, the reference digitalimage including at least one change to the at least one elementcorresponding to how the at least one element will change or has changedduring the time frame; comparing the reference digital image to each ofa plurality of other digital images; based on comparing the referencedigital image to each of the plurality of other digital images,assigning a correlation parameter to each of the plurality of otherdigital images, each correlation parameter indicating how closely arespective other digital image matches the reference digital image; andoutputting for presentation to a user a portion of the plurality ofother digital images having highest correlation parameters.
 2. Themethod of claim 1, wherein: the text string further indicates a name ofa person; and the correlation parameter is assigned to a respectiveother digital image is based, at least in part, on whether the otherdigital image is associated with the name of the person indicated in thetext string.
 3. The method of claim 1, wherein: the text string furtherindicates a name of a location; and the correlation parameter isassigned to a respective other digital image is based, at least in part,on whether the other digital image is associated with location indicatedin the text string.
 4. The method of claim 1, wherein processing thetext string to identify at least the time frame and determine whetherthe time frame is a future time frame or a past time frame comprises thetext string using natural language processing.
 5. The method of claim 1,wherein outputting for presentation to the user the portion of theplurality of other digital images having highest correlation parameterscomprises presenting the portion of the plurality of other digitalimages having highest correlation parameters in an order based thecorrelation parameters.
 6. The method of claim 1, further comprising:assigning a ranking to each of the portion of the plurality of otherdigital images having highest correlation parameters; wherein outputtingfor presentation to the user the portion of the plurality of otherdigital images having highest correlation parameters comprisespresenting the portion of the plurality of other digital images havinghighest correlation parameters in an order based the assigned rankings.7. The method of claim 1, wherein the other digital images are accessedfrom a social networking system.
 8. A system, comprising: a processorprogrammed to initiate executable operations comprising: receiving afirst digital image; receiving a text string indicating at least a timeframe; processing the text string to identify at least the time frameand determine whether the time frame is a future time frame or a pasttime frame; responsive to determining whether the time frame is thefuture time frame or the past time frame, predicting how at least oneelement of the first digital image will change or has changed during thetime frame; responsive to predicting how the at least one element of thefirst digital image will change or has changed during the time frame,generating at least one reference digital image that is a revisedversion of the first digital image, the reference digital imageincluding at least one change to the at least one element correspondingto how the at least one element will change or has changed during thetime frame; comparing the reference digital image to each of a pluralityof other digital images; based on comparing the reference digital imageto each of the plurality of other digital images, assigning acorrelation parameter to each of the plurality of other digital images,each correlation parameter indicating how closely a respective otherdigital image matches the reference digital image; and outputting forpresentation to a user a portion of the plurality of other digitalimages having highest correlation parameters.
 9. The system of claim 8,wherein: the text string further indicates a name of a person; and thecorrelation parameter is assigned to a respective other digital image isbased, at least in part, on whether the other digital image isassociated with the name of the person indicated in the text string. 10.The system of claim 8, wherein: the text string further indicates a nameof a location; and the correlation parameter is assigned to a respectiveother digital image is based, at least in part, on whether the otherdigital image is associated with location indicated in the text string.11. The system of claim 8, wherein processing the text string toidentify at least the time frame and determine whether the time frame isa future time frame or a past time frame comprises the text string usingnatural language processing.
 12. The system of claim 8, whereinoutputting for presentation to the user the portion of the plurality ofother digital images having highest correlation parameters comprisespresenting the portion of the plurality of other digital images havinghighest correlation parameters in an order based the correlationparameters.
 13. The system of claim 8, the executable operations furthercomprising: assigning a ranking to each of the portion of the pluralityof other digital images having highest correlation parameters; whereinoutputting for presentation to the user the portion of the plurality ofother digital images having highest correlation parameters comprisespresenting the portion of the plurality of other digital images havinghighest correlation parameters in an order based the assigned rankings.14. The system of claim 8, wherein the other digital images are accessedfrom a social networking system.
 15. A computer program productcomprising a computer readable storage medium having program code storedthereon, the program code executable by a processor to perform a methodcomprising: receiving, by the processor, a first digital image;receiving, by the processor, a text string indicating at least a timeframe; processing, by the processor, the text string to identify atleast the time frame and determine whether the time frame is a futuretime frame or a past time frame; responsive to determining whether thetime frame is the future time frame or the past time frame, predicting,by the processor, how at least one element of the first digital imagewill change or has changed during the time frame; responsive topredicting how the at least one element of the first digital image willchange or has changed during the time frame, generating, by theprocessor, at least one reference digital image that is a revisedversion of the first digital image, the reference digital imageincluding at least one change to the at least one element correspondingto how the at least one element will change or has changed during thetime frame; comparing, by the processor, the reference digital image toeach of a plurality of other digital images; based on comparing thereference digital image to each of the plurality of other digitalimages, assigning, by the processor, a correlation parameter to each ofthe plurality of other digital images, each correlation parameterindicating how closely a respective other digital image matches thereference digital image; and outputting, by the processor, forpresentation to a user a portion of the plurality of other digitalimages having highest correlation parameters.
 16. The computer programproduct of claim 15, wherein: the text string further indicates a nameof a person; and the correlation parameter is assigned to a respectiveother digital image is based, at least in part, on whether the otherdigital image is associated with the name of the person indicated in thetext string.
 17. The computer program product of claim 15, wherein: thetext string further indicates a name of a location; and the correlationparameter is assigned to a respective other digital image is based, atleast in part, on whether the other digital image is associated withlocation indicated in the text string.
 18. The computer program productof claim 15, wherein processing the text string to identify at least thetime frame and determine whether the time frame is a future time frameor a past time frame comprises the text string using natural languageprocessing.
 19. The computer program product of claim 15, whereinoutputting for presentation to the user the portion of the plurality ofother digital images having highest correlation parameters comprisespresenting the portion of the plurality of other digital images havinghighest correlation parameters in an order based the correlationparameters.
 20. The computer program product of claim 15, the methodfurther comprising: assigning a ranking to each of the portion of theplurality of other digital images having highest correlation parameters;wherein outputting for presentation to the user the portion of theplurality of other digital images having highest correlation parameterscomprises presenting the portion of the plurality of other digitalimages having highest correlation parameters in an order based theassigned rankings.